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ABSTRACT 



A video encoder and decoder are provided for encoding and 
decoding video signal data for an image block and a par- 
ticular reference picture index to predict the image block, 
where the encoder includes a reference picture weighting 
factor selector having an output indicative of a weighting 
factor corresponding to the particular reference picture 
index, a multiplier in signal communication with the refer- 
ence picture weighting factor selector for providing a 
weighted version of the reference picture, and a motion 
estimator in signal communication with the multiplier for 
providing motion vectors corresponding to the weighted 
version of the reference picture; and the corresponding 
decoder a reference picture weighting factor unit having an 
output for determining a weighting factor corresponding to 
the particular reference picture index. 

16 Claims, 5 Drawing Sheets 
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MOTION ESTIMATION WITH WEIGHTING 
PREDICTION 

CROSS-REFERENCE TO RELATED 
APPLICATIONS 

This application claims the benefit of U.S. Provisional 
Patent Application Ser. No. 60/395,874, entitled "Motion 
Estimation With Weighting Prediction" and filed Jul. 15, 
2002, which is incorporated by reference herein in its 
entirety. In addition, this application claims the benefit of 
U.S. Provisional Patent Application Ser. No. 60/395,843, 
entitled "Adaptive Weighting Of Reference Pictures In 
Video CODEC" also filed Jul. 15, 2002, which is incorpo- 
rated by reference herein in its entirety. 

FIELD OF THE INVENTION 

The present invention is directed towards video encoders 
and decoders, and in particular, towards integrated motion 
estimation with weighting prediction in video encoders and 
decoders. 

BACKGROUND OF THE INVENTION 

Video data is generally processed and transferred in the 
form of bit streams. Typical video compression coders and 
decoders ("CODECs") gain much of their compression 
efficiency by forming a reference picture prediction of a 
picture to be encoded, and encoding the difference between 
the current picture and the prediction. The more closely that 
the prediction is correlated with the current picture, the 
fewer bits that are needed to compress that picture, thereby 
increasing the efficiency of the process. Thus, it is desirable 
for the best possible reference picture prediction to be 
formed. 

In many video compression standards, including Moving 
Picture Experts Group ("MPEG")-1, MPEG-2 and MPEG-4, 
a motion compensated version of a previous reference 
picture is used as a prediction for the current picture, and 
only the difference between the current picture and the 
prediction is coded. When a single picture prediction ("P" 
picture) is used, the reference picture is not scaled when the 
motion compensated prediction is formed. When bidirec- 
tional picture predictions ("B" pictures) are used, interme- 
diate predictions are formed from two different pictures, and 
then the two intermediate predictions are averaged together, 
using equal weighting factors of (Vi, Vi) for each, to form a 
single averaged prediction. In these MPEG standards, the 
two reference pictures are always one each from the forward 
direction and the backward direction for B pictures. 

SUMMARY OF THE INVENTION 

These and other drawbacks and disadvantages of the prior 
art are addressed by a system and method for integrated 
motion estimation with weighting prediction in video encod- 
ers and decoders. 

A video encoder and decoder are provided for encoding 
and decoding video signal data for an image block and a 
particular reference picture index to predict the image block, 
where the encoder includes a reference picture weighting 
factor selector having an output indicative of a weighting 
factor corresponding to the particular reference picture 
index, a multiplier in signal communication with the refer- 
ence picture weighting factor selector for providing a 
weighted version of the reference picture, and a motion 
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estimator in signal communication with the multiplier for 
providing motion vectors corresponding to the weighted 
version of the reference picture; and the corresponding 
decoder a reference picture weighting factor unit having an 

5 output for determining a weighting factor corresponding to 
the particular reference picture index. 

A corresponding method for encoding video signal data 
for an image block includes receiving a substantially uncom- 
pressed image block, assigning a weighting factor for the 

10 image block corresponding to a particular reference picture, 
weighting the reference picture by the weighting factor, 
computing motion vectors corresponding to the difference 
between the image block and the weighted reference picture, 
motion compensating the weighted reference picture in 

15 correspondence with the motion vectors, refining the 
weighting factor selection in response to the motion com- 
pensated weighted reference picture, motion compensating 
the original unweighted reference picture in correspondence 
with the motion vectors, multiplying the motion compen- 

20 sated original reference picture by the assigned weighting 
factor to form a weighted motion compensated reference 
picture, subtracting the weighted motion compensated ref- 
erence picture from the substantially uncompressed image 
block, and encoding a signal indicative of the difference 

25 between the substantially uncompressed image block and 
the weighted motion compensated reference picture. 

These and other aspects, features and advantages of the 
present invention will become apparent from the following 
description of exemplary embodiments, which is to be read 

30 in connection with the accompanying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The present invention incorporates integrated motion esti- 
35 mation with weighting prediction in video encoders and 
decoders in accordance with the following exemplary fig- 
ures, in which: 

FIG. 1 shows a block diagram for a standard video 
encoder; 

40 FIG. 2 shows a block diagram for a video encoder with 

reference picture weighting; 

FIG. 3 shows a block diagram for a video encoder with 

integrated motion estimation and weighting prediction in 

accordance with the principles of the present invention; 
45 FIG. 4 shows a block diagram for a standard video 

decoder; 

FIG. 5 shows a block diagram for a video decoder with 
adaptive bi-prediction; 

FIG. 6 shows a flowchart for an encoding process in 
50 accordance with the principles of the present invention; and 

FIG. 7 shows a flowchart for a decoding process in 
accordance with the principles of the present invention. 

DETAILED DESCRIPTION OF PREFERRED 
55 EMBODIMENTS 

An efficient process is provided for integrated motion 
vector estimation and adaptive reference picture weighting 
factor selection. An iterative process is used where an initial 

60 weighting factor is estimated and used in the motion esti- 
mation process. The weighting factor estimate is refined 
based on the results of the motion estimation process. When 
weighting factors are used in encoding, a video encoder 
determines both weighting factors and motion vectors, but 

65 the best choice for each of these depends on the other. 
Motion estimation is typically the most computationally 
intensive part of a digital video compression encoder. 
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In some video sequences, in particular those with fading, 
the current picture or image block to be coded is more 
strongly correlated to a reference picture scaled by a weight- 
ing factor than to the reference picture itself Video 
CODECs without weighting factors applied to reference 5 
pictures encode fading sequences very inefficiently. 

In the proposed Joint Video Team ("JVT") video com- 
pression standard, each P picture can use multiple reference 
pictures to form a picture's prediction, but each individual 
motion block or 8x8 region of a macroblock uses only a 10 
single reference picture for prediction. In addition to coding 
and transmitting the motion vectors, a reference picture 
index is transmitted for each motion block or 8x8 region, 
indicating which reference picture is used. A limited set of 
possible reference pictures is stored at both the encoder and 15 
decoder, and the number of allowable reference pictures is 
transmitted. 

In the JVT standard, for bi-predictive pictures (also called 
"B" pictures), two predictors are formed for each motion 
block or 8x8 region, each of which can be from a separate 20 
reference picture, and the two predictors are averaged 
together to form a single averaged predictor. For bi-predic- 
tively coded motion blocks, the reference pictures can both 
be from the forward direction, both be from the backward 
direction, or one each from the forward and backward 25 
directions. Two lists are maintained of the available refer- 
ence pictures that may used for prediction. The two refer- 
ence pictures are referred to as the list 0 and list 1 predictors. 
An index for each reference picture is coded and transmitted, 
ref_idx_l and ref_idx_ll, for the list 0 and list 1 reference 30 
pictures, respectively. Joint Video Team ("JVT") bi-predic- 
tive or "B" pictures shall allow adaptive weighting between 
the two predictions, i.e., 



Pred=[(PO)*(PredO)]+[(Pl)*(Predl)]+A 



35 



where P0 and PI are weighting factors, PredO and Predl are 
the reference picture predictions for list 0 and list 1 respec- 
tively, and D is an offset. 

Two methods have been proposed for indication of 40 
weighting factors. In the first, the weighting factors are 
determined by the directions that are used for the reference 
pictures. In this method, if the ref_idx_10 index is less than 
or equal to ref_idx_ll, weighting factors of (Vi, Vi) are used, 
otherwise (2, -1) factors are used. 45 

In the second method, any number of weighting factors is 
transmitted for each slice. Then a weighting factor index is 
transmitted for each motion block or 8x8 region of a 
macroblock that uses bidirectional prediction. The decoder 
uses the received weighting factor index to choose the 50 
appropriate weighting factor, from the transmitted set, to use 
when decoding the motion block or 8x8 region. For 
example, if three weighting factors were sent at the slice 
layer, they would correspond to weight factor indices 0, 1 
and 2, respectively. 55 

The following description merely illustrates the principles 
of the invention. It will thus be appreciated that those skilled 
in the art will be able to devise various arrangements that, 
although not explicitly described or shown herein, embody 
the principles of the invention and are included within its 60 
spirit and scope. Furthermore, all examples and conditional 
language recited herein are principally intended expressly to 
be only for pedagogical purposes to aid the reader in 
understanding the principles of the invention and the con- 
cepts contributed by the inventor to furthering the art, and 65 
are to be construed as being without limitation to such 
specifically recited examples and conditions. Moreover, all 



statements herein reciting principles, aspects, and embodi- 
ments of the invention, as well as specific examples thereof, 
are intended to encompass both structural and functional 
equivalents thereof. Additionally, it is intended that such 
equivalents include both currently known equivalents as 
well as equivalents developed in the future, i.e., any ele- 
ments developed that perform the same function, regardless 
of structure. 

Thus, for example, it will be appreciated by those skilled 
in the art that the block diagrams herein represent conceptual 
views of illustrative circuitry embodying the principles of 
the invention. Similarly, it will be appreciated that any flow 
charts, flow diagrams, state transition diagrams, 
pseudocode, and the like represent various processes which 
may be substantially represented in computer readable 
media and so executed by a computer or processor, whether 
or not such computer or processor is explicitly shown. 

The functions of the various elements shown in the figures 
may be provided through the use of dedicated hardware as 
well as hardware capable of executing software in associa- 
tion with appropriate software. When provided by a proces- 
sor, the functions may be provided by a single dedicated 
processor, by a single shared processor, or by a plurality of 
individual processors, some of which may be shared. More- 
over, explicit use of the term "processor" or "controller" 
should not be construed to refer exclusively to hardware 
capable of executing software, and may implicitly include, 
without limitation, digital signal processor ("DSP") hard- 
ware, read-only memory ("ROM") for storing software, 
random access memory ("RAM"), and non-volatile storage. 
Other hardware, conventional and/or custom, may also be 
included. Similarly, any switches shown in the figures are 
conceptual only. Their function may be carried out through 
the operation of program logic, through dedicated logic, 
through the interaction of program control and dedicated 
logic, or even manually, the particular technique being 
selectable by the implementer as more specifically under- 
stood from the context. 

In the claims hereof any element expressed as a means for 
performing a specified function is intended to encompass 
any way of performing that function including, for example, 
a) a combination of circuit elements that performs that 
function or b) software in any form, including, therefore, 
firmware, microcode or the like, combined with appropriate 
circuitry for executing that software to perform the function. 
The invention as defined by such claims resides in the fact 
that the functionalities provided by the various recited 
means are combined and brought together in the manner 
which the claims call for. Applicant thus regards any means 
that can provide those functionalities as equivalent to those 
shown herein. 

As shown in FIG. 1, a standard video encoder is indicated 
generally by the reference numeral 100. An input to the 
encoder 100 is connected in signal communication with a 
non-inverting input of a summing junction 110. The output 
of the summing junction 110 is connected in signal com- 
munication with a block transform function 120. The trans- 
form 120 is connected in signal communication with a 
quantizer 130. The output of the quantizer 130 is connected 
in signal communication with a variable length coder 
("VLC") 140, where the output of the VLC 140 is an 
externally available output of the encoder 100. 

The output of the quantizer 130 is further connected in 
signal communication with an inverse quantizer 150. The 
inverse quantizer 150 is connected in signal communication 
with an inverse block transformer 160, which, in turn, is 
connected in signal communication with a reference picture 
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store 170. A first output of the reference picture store 170 is 
connected in signal communication with a first input of a 
motion estimator 180. The input to the encoder 100 is further 
connected in signal communication with a second input of 
the motion estimator 180. The output of the motion estima- 
tor 180 is connected in signal communication with a first 
input of a motion compensator 190. A second output of the 
reference picture store 170 is connected in signal commu- 
nication with a second input of the motion compensator 190. 
The output of the motion compensator 190 is connected in 
signal communication with an inverting input of the sum- 
ming junction 110. 

Turning to FIG. 2, a video encoder with reference picture 
weighting is indicated generally by the reference numeral 
200. An input to the encoder 200 is connected in signal 
communication with a non-inverting input of a summing 
junction 210. The output of the summing junction 210 is 
connected in signal communication with a block transformer 
220. The transformer 220 is connected in signal communi- 
cation with a quantizer 230. The output of the quantizer 230 
is connected in signal communication with a VLC 240, 
where the output of the VLC 440 is an externally available 
output of the encoder 200. 

The output of the quantizer 230 is further connected in 
signal communication with an inverse quantizer 250. The 
inverse quantizer 250 is connected in signal communication 
with an inverse block transformer 260, which, in turn, is 
connected in signal communication with a reference picture 
store 270. A first output of the reference picture store 270 is 
connected in signal communication with a first input of a 
reference picture weighting factor assigner 272. The input to 
the encoder 200 is further connected in signal communica- 
tion with a second input of the reference picture weighting 
factor assigner 272. The output of the reference picture 
weighting factor assigner 272, which is indicative of a 
weighting factor, is connected in signal communication with 
a first input of a motion estimator 280. A second output of 
the reference picture store 270 is connected in signal com- 
munication with a second input of the motion estimator 280. 

The input to the encoder 200 is further connected in signal 
communication with a third input of the motion estimator 
280. The output of the motion estimator 280, which is 
indicative of motion vectors, is connected in signal commu- 
nication with a first input of a motion compensator 290. A 
third output of the reference picture store 270 is connected 
in signal communication with a second input of the motion 
compensator 290. The output of the motion compensator 
290, which is indicative of a motion compensated reference 
picture, is connected in signal communication with a first 
input of a multiplier 292. The output of the reference picture 
weighting factor assigner 272, which is indicative of a 
weighting factor, is connected in signal communication with 
a second input of the multiplier 292. The output of the 
multiplier 292 is connected in signal communication with an 
inverting input of the summing junction 210. 

Turning now to FIG. 3, a video encoder with integrated 
motion estimation and weighting prediction is indicated 
generally by the reference numeral 300. An input to the 
encoder 300 is connected in signal communication with a 
non- inverting input of a summing junction 310. The output 
of the summing junction 310 is connected in signal com- 
munication with a block transformer 320. The transformer 
320 is connected in signal communication with a quantizer 
330. The output of the quantizer 330 is connected in signal 
communication with a VLC 340, where the output of the 
VLC 440 is an externally available output of the encoder 
300. 
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The output of the quantizer 330 is further connected in 
signal communication with an inverse quantizer 350. The 
inverse quantizer 350 is connected in signal communication 
with an inverse block transformer 360, which, in turn, is 

5 comiected in signal communication with a reference picture 
store 370. A first output of the reference picture store 370 is 
comiected in signal communication with a first input of a 
reference picture weighting factor selector 372. The input to 
the encoder 300 is further connected in signal communica- 

10 tion with a second input of the reference picture weighting 
factor selector 372 to provide the current picture to the 
selector. The output of the reference picture weighting factor 
selector 372, which is indicative of a weighting factor, is 
connected in signal communication with a first input of a 

15 multiplier 374. A second input of the multiplier 374 is 
connected in signal communication with the reference pic- 
ture output of the reference picture store 370. It should be 
noted that although shown simply as a multiplier 374, other 
types of weighting factor applicators may be constructed 

20 other than a multiplier, as would be apparent to those of 
ordinary skill in the art, all of which are contemplated within 
the spirit and scope of the invention. 

The output of the multiplier 374 is connected in signal 
communication with a weighted reference picture store 376. 

25 The output of the weighted reference picture store 376 is 
connected in signal communication with a first input of a 
motion estimator 380 for providing a weighted reference 
picture. The output of the motion estimator 380 is connected 
in signal communication with a first motion compensator 

30 382 for providing motion vectors. The output of the motion 
estimator 380 is further connected in signal communication 
with a first input of a second motion compensator 390. A 
second output of the weighted reference picture store 376 is 
connected in signal communication with a second input of 

35 the first motion compensator 382. 

The output of the first motion compensator 382, which is 
indicative of a weighted motion compensated reference 
picture, is connected in signal communication with a first 
input of an absolute difference generator 384. The input to 

40 the encoder 300, which is the current picture, is further 
connected in signal communication with a second input of 
the absolute difference generator 384. The output of the 
absolute difference function 384 is connected in signal 
communication with a third input of the reference picture 

45 weighting factor selector 372. 

A third output of the reference picture store 370 is 
connected in signal communication with a second input of 
the second motion compensator 390. The output of the 
second motion compensator 390, which is indicative of a 

50 motion compensated reference picture, is connected in sig- 
nal communication with a first input of a multiplier 392. The 
output of the reference picture weighting factor selector 372, 
which is indicative of a weighting factor, is connected in 
signal communication with a second input of the multiplier 

55 392. The output of the multiplier 392 is connected in signal 
communication with an inverting input of the summing 
junction 310. 

As shown in FIG. 4 a standard video decoder is indicated 
generally by the reference numeral 400. The video decoder 

60 400 includes a variable length decoder ("VLD") 410 con- 
nected in signal communication with an inverse quantizer 
420. The inverse quantizer 420 is connected in signal 
communication with an inverse transformer 430. The 
inverse transformer 430 is connected in signal communica- 

65 tion with a first input terminal of an adder or summing 
junction 440, where the output of the summing junction 440 
provides the output of the video decoder 400. The output of 
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the summing junction 440 is connected in signal commu- compensated reference picture, mcref, by applying the MVs 

nication with a reference picture store 450. The reference to ref. The block 630 passes control to a function block 632 

picture store 450 is connected in signal communication with that refines the weighting factor estimate by setting w equal 

a motion compensator 460, which is connected in signal to avg(cur)/avg(mcref). The block 632 passes control back 

communication with a second input terminal of the summing 5 to the function block 616 for further processing. Thus, the 

junction 440. decision to further refine the weighting factor is based on 

Turning to FIG. 5 a video decoder with adaptive bi- comparing a difference measure to a threshold or tolerance, 

prediction is indicated generally by the reference numeral Turning now to FIG. 7 an exemplary process for decoding 

500. The video decoder 500 includes a VLD 510 connected video signal data for an image block is indicated generally 

in signal communication with an inverse quantizer 520. The 10 by the reference numeral 700. The process includes a start 

inverse quantizer 520 is connected in signal communication block 710 that passes control to an input block 712. The 

with an inverse transformer 530. The inverse transformer input block 712 receives the image block compressed data, 

530 is connected in signal communication with a first input and passes control to an input block 714. The input block 

terminal of a summing junction 540, where the output of the 714 receives at least one reference picture index with the 

summing junction 540 provides the output of the video 15 data for the image block, each reference picture index 

decoder 500. The output of the summing junction 540 is corresponding to a particular reference picture. The input 

connected in signal communication with a reference picture block 714 passes control to a function block 716, which 

store 550. The reference picture store 550 is connected in determines a weighting factor corresponding to each of the 

signal communication with a motion compensator 560, received reference picture indices, and passes control to an 

which is comiected in signal communication with a first 20 optional function block 717. The optional function block 

input of a multiplier 570. 717 determines an offset corresponding to each of the 

The VLD 510 is further connected in signal communica- received reference picture indices, and passes control to a 
tion with a reference picture weighting factor lookup 580 for function block 718. The function block 718 retrieves a 
providing an adaptive bi-prediction ("ABP") coefficient reference picture corresponding to each of the received 
index to the lookup 580. A first output of the lookup 580 is 25 reference picture indices, and passes control to a function 
for providing a weighting factor, and is connected in signal block 720. The function block 720, in turn, motion corn- 
communication to a second input of the multiplier 570. The pensates the retrieved reference picture, and passes control 
output of the multiplier 570 is connected in signal commu- to a function block 722. The function block 722 multiplies 
nication to a first input of a summing junction 590. A second the motion compensated reference picture by the corre- 
output of the lookup 580 is for providing an offset, and is 30 sponding weighting factor, and passes control to an optional 
connected in signal communication to a second input of the function block 723. The optional function block 723 adds 
summing junction 590. The output of the summing junction the motion compensated reference picture to the correspond- 
590 is connected in signal communication with a second ing offset, and passes control to a function block 724. The 
input terminal of the summing junction 540. function block 724, in turn, forms a weighted motion 

Turning now to FIG. 6, a motion vector and weighting 35 compensated reference picture, and passes control to an end 

factor determination process is indicated generally by the block 726. 

reference numeral 600. Here, a function block 610 finds the In the present exemplary embodiment, for each coded 
initial weighting factor estimate for the current picture or picture or slice, a weighting factor is associated with each 
image block ("cur") and reference picture ("ref) by com- allowable reference picture that blocks of the current picture 
puting the weighting factor "w"=avg(cur)/avg(ref). The 40 can be encoded with respect to. When each individual block 
block 610 passes control to a decision block 612 that in the current picture is encoded or decoded, the weighting 
determines whether the weighting factor w is greater than a factor(s) and offset(s) that correspond to its reference picture 
threshold value Tl and less than a threshold value T2. If w indices are applied to the reference prediction to form a 
is between Tl and T2, control is passed to a return block 614 weight predictor. All blocks in the slice that are coded with 
and w=l is used as the initial weighting factor. If w is not 45 respect to the same reference picture apply the same weight- 
between Tl and T2, control is passed to a function block 616 ing factor to the reference picture prediction, 
that applies the weighting factor w to the reference picture Whether or not to use adaptive weighting when coding a 
to form a weighted reference picture wref. The block 616 picture can be indicated in the picture parameter set or 
passes control to a function block 618 to perform motion sequence parameter set, or in the slice or picture header. For 
estimation by finding motion vectors ("MVs") using the 50 each slice or picture that uses adaptive weighting, a weight- 
weighted reference picture wref. The block 618 passes ing factor may be transmitted for each of the allowable 
control to a function block 620 that forms a motion com- reference pictures that may be used for encoding this slice or 
pensated weighted reference picture, mcwref, by applying picture. The number of allowable reference pictures is 
the MVs to wref. The block 620 passes control to a function transmitted in the slice header. For example, if three refer- 
block 622 that calculates a difference measure, diff, where 55 ence pictures can be used to encode the current slice, up to 
diff equals the absolute value of the sum of the pixel three weighting factors are transmitted, and they are asso- 
differences between cur and wmcref. ciated with the reference picture with the same index. 

The block 622 passes control to a decision block 624 that If no weighting factors are transmitted, default weights 

determines whether diff is greater than the previous best diff. are used. In one embodiment of the present invention, 

If diff is greater than the previous best diff, control is passed 60 default weights of ( 1 /2, 1 /2) are used when no weighting 

to a return block 626, which uses the previous best diff. If factors are transmitted. The weighting factors may be trans- 

diff is not greater than the previous best diff, control is mitted using either fixed or variable length codes, 

passed to a decision block 628 that determines whether diff Unlike typical systems, each weighting factor that is 

is less than a threshold T. If diff is less than the threshold T, transmitted with each slice, block or picture corresponds to 

then control is passed to a return block 634 that uses the 65 a particular reference picture index. Previously, any set of 

current estimates. If diff is not less than the threshold T, then weighting factors transmitted with each slice or picture was 

control is passed to a function block 630 that forms a motion not associated with any particular reference pictures. 
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Instead, an adaptive bi-prediction weighting index was Thus, a weighting factor is applied to the reference picture 

transmitted for each motion block or 8x8 region to select prediction of a video compression encoder and decoder that 

which of the weighting factors from the transmitted set was uses multiple reference pictures. The weighting factor 

to be applied for that particular motion block or 8x8 region. adapts for individual motion blocks within a picture, based 

In the instant embodiment of the present invention, the 5 on the reference picture index that is used for that motion 
weighting factor index for each motion block or 8x8 region block. Because the reference picture index is already trans- 
is not explicitly transmitted. Instead, the weighting factor m itted in the compressed video bitstream, the additional 
that is associated with the transmitted reference picture overhead to adapt the weighting factor on a motion block 
index is used. This dramatically reduces the amount of basis is dramatically reduced. All motion blocks that are 
overhead in the transmitted bitstream to allow adaptive 10 coded with respect to the same reference picture apply the 
weighting of reference pictures. same weighting factor to the reference picture prediction. 

This system and technique may be applied to either Motion estimation techniques have been widely studied. 

Predictive T pictures, which are encoded with a single For each motion block of a picture being coded7 a motion 

predictor, or to Bi-predictive "B" pictures, which are vector is chosen that represents a displacement of the motion 

encoded with two predictors. The decoding processes, which 15 Mock from a reference picture. In an exhaustive search 

are present in both encoder and decoders, are described met hod within a search region, every displacement within a 

below for the P and B picture cases. Alternatively, this pr e-determined range of offsets relative to the motion block 

technique may also be applied to coding systems using the position is tested. The test includes calculating the sum of 

concepts similar to I, B, and P pictures. the abso lute difference ("SAD") or mean squared error 

The same weighting factors can be used for single direc- 20 (« MSE ») of each pixel in the mot ion block in the current 

tional prediction in B pictures and for bi-directional predic- picture with the displaced motion block in a reference 

tion in B pictures. When a single predictor is used for a picture. The offset with the lowest SAD or MSE is selected 

macroblock, in P pictures or for single directional prediction as the mot ion vector. Numerous variations on this technique 

in B pictures, a single reference picture index is transmitted have bee n proposed, such as three-step search and rate- 

for the block. After the decoding process step of motion 25 distortion optimized motion estimation, all of which include 

compensation produces a predictor, the weighting factor is the step 0 f computing the SAD or MSE of the current motion 

applied to predictor. The weighted predictor is then added to block with a displaced motion block in a reference picture. 

the coded residual, and clipping is performed on the sum, to ^ + +• i + j + • • +• + a 

r . , i j . £ / , i i • ™ • Computational costs of determining motion vectors and 

lorm the decoded picture, r or use lor blocks in P pictures or A # . . . 1 ■ + i , , 

„ t t 1 . ^ y , 1 i- /x 1- • i on adaptive reference picture weighting factors can be reduced 

lor blocks in B pictures that use onlv list 0 prediction, the 30 i • i -r +n 1 + - 

wei hted redictor is formed as* using an iterative process, while still selecting motion 

° P c vectors and weighting factors that are able to achieve high 

Pred=wo*Predo+DO (l) compression efficiencies. An exemplary embodiment 

motion vector and weighting factor determination process is 

where WO is the weighting factor associated with the list 0 35 described assuming that a single weighting factor is applied 

reference picture, DO is the offset associated with the list 0 to the entire reference picture, although application of the 

reference picture, and PredO is the motion-compensated principles of the present invention are not limited to such a 

prediction block from the list 0 reference picture. case. The process could also be applied over smaller regions 

For use for blocks in B pictures that use only list 0 of the picture, such as slices, for example. In addition, 

prediction, the weighted predictor is formed as: 4Q although one exemplary embodiment of the invention is 

Pred=W\*Pred\+D\ (2) described as using only a single reference picture, the 

present invention may also be applied to multiple reference 

where Wl is the weighting factor associated with the list 1 P icture Prediction and to bi-predictive pictures, 

reference picture, DO is the offset associated with the list 1 Calculation of the motion vector for a motion block can 

reference picture, and Predl is the motion-compensated 45 typically best be done when the weighting factor to be used 

prediction block from the list 1 reference picture. is known. In an exemplary embodiment, an estimate of the 

The weighted predictors may be clipped to guarantee that weighting factor is formed, using the reference picture and 

the resulting values will be within the allowable range of the current picture pixel values. The weighting factor may be 

pixel values, typically 0 to 255. The precision of the mul- limited to a number of bits of resolution. If the weighting 

tiplication in the weighting formulas may be limited to any 50 factor is ver Y close to 1, there is no need to consider the 

pre-determined number of bits of resolution. weighting factor in the motion estimation process, and 

In the bi-predictive case, reference picture indexes are normal motion estimation can be done with the weighting 

transmitted for each of the two predictors. Motion compen- factor assumed to be equal to 1. Otherwise, the weighting 

sation is performed to form the two predictors. Each pre- factor estimate is applied to the reference picture. Motion 

dictor uses the weighting factor associated with its reference 55 estimation is then performed using any method which cal- 

picture index to form two weighted predictors. The two culates SAD or MSE, but with the SAD or MSE calculation 

weighted predictors are then averaged together to form an performed between the current picture motion block and the 

averaged predictor, which is then added to the coded displaced motion block in the weighted version of the 

residual. reference picture, rather than the un- weighted reference 

For use for blocks in B pictures that use list 0 and list 1 60 picture. After motion vectors have been selected, the esti- 

predictions, the weighted predictor is formed as: mat i«n of the weighting factor can be refined, if necessary. 



Pred=(P0 *Pred0+D0+Pl *Predl+Dl )/2 (3) 



The current motion vectors are applied to the weighted 
reference picture to form the weighted, motion compensated 
Clipping may be applied to the weighted predictor or any reference picture. A difference measure between the 
of the intermediate values in the calculation of the weighted 65 weighted, motion compensated reference picture and the 
predictor to guarantee that the resulting values will be within current picture is computed. If the difference measure is 
the allowable range of pixel values, typically 0 to 255. lower than a threshold, or lower than the previous best 
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difference measure, the process is complete, and the current When multiple reference pictures are used to encode a 

candidate motion vectors and weighting factor are accepted. picture, a separate weighting factor can be calculated for 

If the difference measure is higher than some threshold, each reference picture. During motion estimation, a motion 

the weighting factor can be refined. In this case, a motion vector and a reference picture index are selected for each 

compensated but un- weighted reference picture is formed 5 motion block. For each iteration of the process, motion 

based on the current candidate motion vectors. The weight- vectors and weighting factors are found for each reference 

ing factor estimate is refined using the motion compensated picture. 

reference picture and the current picture, rather than using In a preferred embodiment, during motion estimation, the 

the un-compensated reference picture, as was done in form- best reference picture for a given motion block is deter- 

ing the initial estimate of the weighting factor. 10 mined. Calculation of the difference measure is done sepa- 

The selection process proceeds to iterate, applying the rately for each reference picture, with only those motion 

newly refined weighting factor to the reference picture to blocks that use that reference picture being used in the 

form the weighted reference picture. The iterative process calculation. Refinement of the weighting factor estimate for 

continues until the difference measure is equal or higher than a given reference picture also uses only those motion blocks 

a previous best difference measure, or lower than a thresh- 15 that are coded using that reference picture. For bi-predictive 

old, or alternatively, until a defined number of cycles has coding, weighting factors and motion vectors can be deter- 

been completed. If the difference measure of the current mined separately for each of the two predictions, which will 

iteration is higher than for the best previous iteration, the be averaged together to form the averaged prediction, 

weighting factor and motion vectors for the best previous The principles of the present invention can be applied to 

iteration are used. If the difference measure of the current 20 many different types of motion estimation algorithms. When 

iteration is less than a threshold, the current weighting factor used with hierarchical approaches, the iteration of weighting 

and motion vectors are used. If the maximum number of factor selection and motion vector selection can be used with 

iteration cycles has been completed, the weighting factor any level of the motion estimation hierarchy. For example, 

and motion vectors from the previous iteration that had the the iterative approach could be used with integer picture 

best difference measure are used. 25 element ("pel") motion estimation. After the weighting 

In one embodiment, the initial estimate of the weighting factor and integer motion vectors are found using the 

factor, w, is the ratio between the average value of the pixels provided iterative algorithm, the sub-pel motion vectors may 

in the current picture, cur, divided by the average value of be found without requiring another iteration of the weighting 

the pixels in the reference picture, ref, where: factor selection. 

30 These and other features and advantages of the present 

w-avg(cur)lavg(ref) (4) invention may be readily ascertained by one of ordinary skill 

The refinement estimates are the ratio between the aver- in the pertinent art based on the teachings herein. It is to be 

age of pixels in the current picture and the average of pixels understood that the principles of the present invention may 

in the motion compensated reference picture, mcref, where: be implemented in various forms of hardware, software, 

, x , , A ' 35 firmware, special purpose processors, or combinations 

w=avg(cur)lavg(mcref) (5) thereof 

The difference measure diff is the absolute value of the Most preferably, the principles of the present invention 

average of pixel differences between the current picture, cur, are implemented as a combination of hardware and software, 

and the weighted motion compensated reference picture, Moreover, the software is preferably implemented as an 

wmcref, where: 40 application program tangibly embodied on a program stor- 

diff=\Lcur-wmcref (6) a § e umt * ^ ne application program may be uploaded to, and 

executed by, a machine comprising any suitable architecture. 

In another embodiment, the difference measure is the sum Preferably, the machine is implemented on a computer 

of the absolute differences of the pixels in the current picture platform having hardware such as one or more central 

and in the weighed motion compensated reference picture, 45 processing units ("CPU"), a random access memory 

where: ("RAM"), and input/output ("I/O") interfaces. The computer 

diff=Hcur-wmcrej\ (7) platform may also include an operating system and micro- 
instruction code. The various processes and functions 

When block-based motion estimation is performed, the described herein may be either part of the microinstruction 

same pixel in a reference picture is used for numerous SAD 50 code or part of the application program, or any combination 

calculations. In an exemplary embodiment during the thereof, which may be executed by a CPU. In addition, 

motion estimation process, once a weighting factor has been various other peripheral units may be connected to the 

applied to a pixel in a reference picture, the weighted pixel computer platform such as an additional data storage unit 

is stored, in addition to the normal pixel. The storage may be and a printing unit. 

done either for a region of the picture, or for the entire 55 It is to be further understood that, because some of the 
picture. constituent system components and methods depicted in the 
The weighted reference picture values may be clipped to accompanying drawings are preferably implemented in soft- 
be stored with the same number of bits as an unweighted ware, the actual connections between the system compo- 
reference, such as 8 bits, for example, or may stored using nents or the process function blocks may differ depending 
more bits. If clipping is performed for the motion compen- 60 upon the manner in which the present invention is pro- 
sation process, which is more memory efficient, the weight- grammed. Given the teachings herein, one of ordinary skill 
ing factor is reapplied to the reference picture for the actual in the pertinent art will be able to contemplate these and 
selected motion vector, the difference is calculated using similar implementations or configurations of the present 
additional bits, and the clipping is performed after the invention. 

difference in order to avoid mismatch with a decoder, which 65 Although the illustrative embodiments have been 
mi glit otherwise occur if the decoder does not perform described herein with reference to the accompanying draw- 
clipping after the weighting factor is applied. ings, it is to be understood that the present invention is not 
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limited to those precise embodiments, and that various 
changes and modifications may be effected therein by one of 
ordinary skill in the pertinent art without departing from the 
scope or spirit of the present invention. All such changes and 
modifications are intended to be included within the scope of 5 
the present invention as set forth in the appended claims. 
What is claimed is: 

1. A method for encoding video signal data for an image 
block, the method comprising: 

receiving a substantially uncompressed image block; 10 

assigning a weighting factor for the image block corre- 
sponding to a particular reference picture; 

weighting the reference picture by the weighting factor; 

computing motion vectors corresponding to the difference 
between the image block and the weighted reference 15 
picture; and 

motion compensating the weighted reference picture in 

correspondence with the motion vectors; 
wherein bi-predictive picture predictors are used, and the 

method further comprises: 20 
assigning a second weighting factor for the image block 

corresponding to a second particular reference picture; 
weighting the second reference picture by the second 

weighting factor; 
computing second motion vectors corresponding to the 25 

difference between the image block and the second 

weighted reference picture; and 
motion compensating the second weighted reference pic- 
ture in correspondence with the second motion vectors. 

2. A method as defined in claim 1, further comprising: 30 
motion compensating the original unweighted reference 

picture in correspondence with the motion vectors; 
multiplying the motion compensated original reference 

picture by the assigned weighting factor to form a 

weighted motion compensated reference picture; 35 
subtracting the weighted motion compensated reference 

picture from the substantially uncompressed image 

block; and 

encoding a signal indicative of the difference between the 
substantially uncompressed image block and the 40 
weighted motion compensated reference picture. 

3. A method as defined in claim 1 wherein computing 
motion vectors comprises: 

testing within a search region for every displacement 
within a pre-determined range of offsets relative to the 45 
image block; 

calculating at least one of the sum of the absolute differ- 
ence and the mean squared error of each pixel in the 
image block with a motion compensated reference 
picture; and 50 

selecting the offset with the lowest sum of the absolute 
difference and mean squared error as the motion vector. 

4. A method as defined in claim 2 the method further 
comprising: 

motion compensating the original unweighted second 55 
reference picture in correspondence with the second 
motion vectors; 

multiplying the motion compensated original second ref- 
erence picture by the assigned second weighting factor 
to form a second weighted motion compensated refer- 60 
ence picture; 

subtracting the second weighted motion compensated 
reference picture from the substantially uncompressed 
image block; and 

encoding a signal indicative of the difference between the 65 
substantially uncompressed image block and the sec- 
ond weighted motion compensated reference picture. 



5. A method as defined in claim 1 wherein the first and 
second particular reference pictures are both from the same 
direction relative to the image block. 

6. A method as defined in claim 1 wherein computing 
motion vectors comprises: 

testing within a search region for every displacement 
within a pre-determined range of offsets relative to the 
image block; 

calculating at least one of the sum of the absolute differ- 
ence and the mean squared error of each pixel in the 
image block with a first motion compensated reference 
picture corresponding to the first predictor; 

selecting an offset with the lowest sum of the absolute 
difference and mean squared error as the motion vector 
for the first predictor; 

calculating at least one of the sum of the absolute differ- 
ence and the mean squared error of each pixel in the 
image block with a second motion compensated refer- 
ence picture corresponding to the second predictor; and 

selecting an offset with the lowest sum of the absolute 
difference and mean squared error as the motion vector 
for the second predictor. 

7. A method as defined in claim 1 wherein weighting the 
reference picture by the weighting factor comprises: 

determining whether the weighting factor is close to about 
1; and 

using the original reference picture as the weighted ref- 
erence picture if the weighting factor is close to about 
1. 

8. A method as defined in claim 1 wherein refining the 
weighting factor selection in response to the motion com- 
pensated weighted reference picture comprises: 

calculating a difference between the image block and the 
motion compensated weighted reference picture; 

comparing the calculated difference to a pre-determined 
tolerance; and 

further refining the weighting factor if the calculated 
difference is outside of the predetermined tolerance. 

9. An apparatus for encoding video signal data for an 
image block, the method comprising: 

means for receiving a substantially uncompressed image 
block; 

means for assigning a weighting factor for the image 
block corresponding to a particular reference picture; 

means for weighting the reference picture by the weight- 
ing factor; 

means for computing motion vectors corresponding to the 
difference between the image block and the weighted 
reference picture; and 

means for motion compensating the weighted reference 
picture in correspondence with the motion vectors; 

wherein bi-predictive picture predictors are used, and the 
apparatus further comprises: 

means for assigning a second weighting factor for the 
image block corresponding to a second particular ref- 
erence picture; 

means for weighting the second reference picture by the 
second weighting factor; 

means for computing second motion vectors correspond- 
ing to the difference between the image block and the 
second weighted reference picture; and 

means for motion compensating the second weighted 
reference picture in correspondence with the second 
motion vectors. 

10. An apparatus as defined in claim 9, further compris- 
ing: 
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means for motion compensating the original unweighted 
reference picture in correspondence with the motion 
vectors; 

means for multiplying the motion compensated original 
reference picture by the assigned weighting factor to 5 
form a weighted motion compensated reference pic- 
ture; 

means for subtracting the weighted motion compensated 
reference picture from the substantially uncompressed 
image block; and 10 

means for encoding a signal indicative of the difference 
between the substantially uncompressed image block 
and the weighted motion compensated reference pic- 
ture. 

11. An apparatus as defined in claim 10, the method 15 
further comprising: 

motion compensating the original unweighted second 
reference picture in correspondence with the second 
motion vectors; 

multiplying the motion compensated original second ref- 20 
erence picture by the assigned second weighting factor 
to form a second weighted motion compensated refer- 
ence picture; 

subtracting the second weighted motion compensated 
reference picture from the substantially uncompressed 25 
image block; and 

encoding a signal indicative of the difference between the 
substantially uncompressed image block and the sec- 
ond weighted motion compensated reference picture. 

12. An apparatus as defined in claim 9 wherein computing 30 
motion vectors comprises: 

testing within a search region for every displacement 
within a pre-determined range of offsets relative to the 
image block; 

calculating at least one of the sum of the absolute differ- 35 
ence and the mean squared error of each pixel in the 
image block with a motion compensated reference 
picture; and 

selecting the offset with the lowest sum of the absolute 
difference and mean squared error as the motion vector. 
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13. An apparatus as defined in claim 9 wherein the first 
and second particular reference pictures are both from the 
same direction relative to the image block. 

14. An apparatus as defined in claim 9 wherein computing 
motion vectors comprises: 

testing within a search region for every displacement 
within a pre-determined range of offsets relative to the 
image block; 

calculating at least one of the sum of the absolute differ- 
ence and the mean squared error of each pixel in the 
image block with a first motion compensated reference 
picture corresponding to the first predictor; 

selecting an offset with the lowest sum of the absolute 
difference and mean squared error as the motion vector 
for the first predictor; 

calculating at least one of the sum of the absolute differ- 
ence and the mean squared error of each pixel in the 
image block with a second motion compensated refer- 
ence picture corresponding to the second predictor; and 

selecting an offset with the lowest sum of the absolute 
difference and mean squared error as the motion vector 
for the second predictor. 

15. An apparatus as defined in claim 9 wherein weighting 
the reference picture by the weighting factor comprises: 

determining whether the weighting factor is close to about 
1; and 

using the original reference picture as the weighted ref- 
erence picture if the weighting factor is close to about 
1. 

16. A method as defined in claim 9 wherein refining the 
weighting factor selection in response to the motion com- 
pensated weighted reference picture comprises: 

calculating a difference between the image block and the 
motion compensated weighted reference picture; 

comparing the calculated difference to a pre-determined 
tolerance; and 

further refining the weighting factor if the calculated 
difference is outside of the predetermined tolerance. 



